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ABSTRACT 

The primary purpose of this study was to investigate 
the effectiveness of two different types of pos t~ ins t rue t i on concept 
interviews: one that did and one that did not embed a concept mapping 
process as means of eliciting students' pos t- ins t rue t i on conceptual 
understandings about the nature of, source of, and problems caused by 
ch 1 o r o f 1 uo r o car bons (CFCs) . The study also tried to determine any 
effect by the addition of the independent variable, student to expert 
Pathfinder Network Similarity Index (PFNSI) on the prediction of the 
criterion measure of accordance (ACCORD). The study also sought to 
elicit students' perceptions of the process instituted in the 
interviews to determine what they know. Data was collected from 
eighth grade physical science students and their teacher using 
open-ended interviews. Findings indicate an interview that embedded a 
concept mapping process (compared to an interview that excluded this 
process) did not affect statistically significant changes in the 
ex t ernal izat i on of students' conceptual understandings. Another 
finding was that PFNSI had predictive validity for performance in the 
interview on the measure of ACCORD and proved to be a reliable 
confirmatory measure of the degree to which students held an ideal 
post-instructional understanding. Contains 67 references. (JRH) 
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Over the past decade, the concept map has emerged as a versatile and promising tool in the area 
of science education, especially in research that seeks to investigate students' conceptual understandings 
(Good, Novak, & Wandersee, 1990; Markham, Minlz.es, & Jones, 1994; Novak, 1990a: Wandersee, 
Mintzes, & Novak, 1994). Concept mapping was developed as a strategy to probe knowledge structures 
of learners (Driver, 1989), and has further utility as a tool for representing and assessing conceptual 
understandings, alternatively termed structural knowledge (Jonassen, Beissner, & Yacci, 1993) or 
cognitive structure (Acton, Johnson, & Goldsmith, 1994). The concept map as a vehicle to represent and 
assess changes in students' understandings about science was pioneered by Novak and colleagues 
(Horton, McConney, Gallo. Woods. Senn. & Hamelin, 1993; Novak, 1990b; Novak & Musonda, 1991: 
Okebukola, 1992). 

Novak and Gowin (1984) have recommended three uses of concept maps in concert with 
interviews to evaluate students' understandings: (a) to inform the development of interview questions, 
(b) to explicate "post hoc" the student understandings data captured by the interview transcripts, and (c) 
to assess student understandings as captured in the interview transcripts. Wandersee et al .( 1 994) cite 
Novak and Gowin as "see[ing] concept maps constructed from taped interviews as an excellent way to 
multiply the power of two tools [concept maps and interviews]” (p. 200). These interview-related 
applications of concept maps have been recommended and applied by several researchers (e.g., Auld, 
1990: Brody, 1991; Heinze-Fry. 1987; Novak & Musonda, 1991; Nhkhleh, 1990; White & Gunstone, 
1992) and are reviewed in detail elsewhere (Rye, 1995). 

Central to the research conducted by this study is the supposition that concept mapping, when 
employed as an integral component of a concept interview' (White & Gunstone, 1992), will facilitate the 
exlernali/.ation of students' understandings. In discussing the dimension of availability of knowledge. 
White (1985) posits that individuals who are equally knowledgeable may differ in their facility to recall 
relevant dements of that knowledge and that "the source of such a difference is an absorbing realm for 

'A paper presented at the 1996 Annual Meeting of the National Association for Research in Science 
Teaching, St. Louis, MO, March 31- April 3. This research was supported in part by the National 
Science Foundation under Grant No. TEP-9 150232, and a grant from The Pennsylvania State University 
College of Education Alumni Society. Any opinions, findings, and conclusions or recommendations 
expressed in this paper are those of the authors and do not necessarily reflect the view's of the National 
Science Foundation or the Alumni Society. 



research . . . if explained, may lead to dramatic improvements in human performance" tp. 56). 
Accordingly, interview-based research that seeks to investigate another individual's understandings must 
recognize a critical and central concern, that being the researcher's ability to externalize and describe 
those particular understandings. The researcher's ability to elicit knowledge is identified widely as a 
problem area (Cohen. 1991; Firlej & Hellens, 1991; LaFrance, 1992; Pidgeon, Turner, & Blockley, 

1991; Swafficld, 1990). In his discussion of methodological difficulties surrounding the interview as a 
means to measure cognitive structure. Shucll (1985) remarks that "students often arc not able to 
articulate what they know" (p. 123). Pidgeon et al. (1991) quotes Polanyi, who contends that "we know 
more than we can say" is a significant social science concern. Knowledge elicitation is complicated 
further by the fact that some knowledge, especially that of experts, is tacit and implicit, making 
verbalization difficult (Cordingley. 1989; Gordon et al., 1993). The previous are not small concerns, 
considering that knowledge elicitation is a prerequisite to representing and assessing an individual's 
understanding (Jonassen et al., 1993). 

The concept map is considered to be a graphic metacognitive tool (Johnson & Thomas, 1992; 
Novak, 1985; Wandersee. 1990; West, Farmer. & Wolf, 1991). As such, the concept map provides an 
external representation of structural knowledge in the form of a two-dimensional semantic network 
(Jonassen, 1996). potentially extends working memory, and encourages critical thinking. Furthermore, 
the concept map as an external representation of structural knowledge serves as a visual image; It may 
trigger recognition, thereby lowering the threshold of availability and increase recall of knowledge held 
in iong term memory (Ausubel et al., 1978). 

Flavell (1985) states that metacognition "includes any knowledge or cognitive activity that takes 
as its object or regulates, any aspect of any cognitive enterprise” (p. 1 16). Metacognition includes 
"strategic actions of the reasoner" (Eylon & Linn, 1988. p. 280), such as questioning, self-checking, and 
generating alternatives (Johnson & Thomas, 1992). The concept map, as a "mindtool" (Jonassen. 1996). 
may facilitate such strategic actions in the interview setting. For example, it may cause students to 
reflect more so on what they know and say, thereby stimulating spread-of-aetivation and leading to 
further recall and elaboration (Gagne. 1985; Johnson & Thomas. 1992; Wandersee et al., 1994), 

A literature review failed to disclose research that investigated the use of the concept map as an 
interview tool to enhance students' recall and elicitation of knowledge, relative to the interview questions 
posed. Wandersee and colleagues (J. Wandersee. personal communication, November 24. 1993) report 
their development of a "coeonstruction of concept map" technique that is deployed in the interview 
setting. This process has potential to increase the validity of concept map representations of students' 
understandings that are externalized through interview questions, as opposed to relying on post hoc 
researcher-constructed concept maps from the student transcripts that repiesent the "researcher’s 
understanding" of the student's understanding 



In the knowledge engineering literature (Smith, 1987), a concept map-like structuie— labeled a 
conceptual graph structure (CGS)--has been employed as a tool in conjuction with interviews to elicit or 
"extract” knowledge from experts (Gordon & Gill, 1989, 1991; Gordon, Schmierer, & Gill, 1993). This 
knowledge engineering process is referred to as conceptual graph analysis. Like concept maps, CGS are 
comprised of nodes connected by labeled links that explicate node interrelationships. Question probes 
are used in concert with the evolving CGS. the latter serving as an auxiliary visual during the interviews 
to facilitate elicitation and verification of the expert’s understandings. Ultimately, the CGS becomes the 
"expert" representation of knowledge, and informs the design of text or instruction, or is encoded as the 
knowledge base in an expert system. CGS and the rela’cd analytical process function to minimize any 
discrepancy or gap between the knowledge engineer's understanding of the expert's knowledge and the 
expert's actual knowledge. Gordon et al. ( 1993) report that the utilization of interview question probes 
in concert with CGS provide a "sensitise means of measuring student knowledge structures" (p. 479). In 
this study, the investigators adapted the conceptual graph analysis process to design a teacher-expert 
concept map, which was employed as a referent to analyze student understandings data elicited in the 
interview setting. 

The teacher-expert concept map also was used to identify central concepts for Pathfinder analysis 
tSchvaneveldt. 1990). Pathfinder Networks are two-dimensional link weighted graphic representations 
of structural knowledge (Jonassen et al.. 1993). Pathfinder Networks are derived through application of 
the Pathfinder scaling algorithm (Goldsmith. Johnson, & Acton. 1991 ) to the relatedness ratings of 
concept pairs-- concept relatedness latittgs alternatively are known as pairwise relatedness ratings 
(Goldsmith et al.. 1991 ) or semantic proximities. The resulting network representations may be 
compared to yield an index of configurational similarity (Goldsmith et al., 1991 ) of each student's to the 
teacher's net — this similarity index alternatively is known as "Closeness" (Acton. Johnson. & Goldsmith. 
1994) and is believed to be an excellent measure of students' structural knowledge (Goldsmith et al.. 
1991; Jonassen, 1991). 

The Pathfinder Network is a relatively new tool in educational research (Goldsmith et al.. 1991). 
This study explored new ground by utilizing an expert concept map to inform Pathfinder analysis .aid in 
examining the predictive validity of the Pathlindei Network similauty index for student performance in 
the interview setting A review of the literature revealed but one study (Wilson. 1994) that has 
combined the concept map and Pathfinder Network in investigating student understandings in science, 
and no studies that apply in concert these tools to data analysis in the manner described herein. Wilson 
employed the Pathfinder settling algorithm in an investigation of the differences in post-instructional 
concept maps ion chemical equilibrium) of high and low achievers in high school chemistry. The 
similarity index of the Pathfinder Networks of the two groups (high and low achievers) was 38. The 
P.ilhfiudei Network of the high achiever's group had the more broad inclusive concepts located centially 
and the more concrete concepts located on the periphery whereas the net of the lower achiever's group 
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placed several ot the less inclusive concepts centrally. The author concluded that their results support 
the claim that "the way know ledge is organized is a major source of the difference between experts and 
novices" (p, 1 1 43). 



Background 

This study emerged from research (Rye. Rubba. & Wiesemnayer. 1994, in press) that was 
conducted as part of a National Science Foundation sponsored Teacher Enhancement Project (Rubha. 
Wiesemnayer. Rye. & Ditty. 199b). This project sought to develop teacher-leaders and currieula to 
further Science- 1 echnology-Socielv (SIS) education in middle/junior high school science. The STS 
issues of focus were global wanning and others encompassed by global atmospheric change (Ennis A 
Marcus. 1994: Houghton. Callendai. Varne. 1992). 

During the early stages of the project, teams of teacher-participants developed STS-global 
warming investigation and action units (Rubha 6c Wiesemnayer. 1985). Each unit contained lessons 
that presented STS foundations and a general awareness of STS issues prior to lessons that honed in on 
investigating and taking action on global warming. As part of formative evaluation of these units, 
project staff conducted semi-structured interviews (Merriam, 1988; Novak 6c Gowin. 1984) with grades 
5-9 students after they had completed the STS-global w arming instruction. These interviews included 
questions to elicit the students' post-instructional understandings about the nature, cause, and resolution 
ot global warming Analysis of the interview transcripts incorporated the concept man tool as an 
"expert" telerent structure (Jonassen el al.. 1993 ; Loinask. Baron. & Grieg. 1993 ) derived from 
document analysis (Gordon el ah. 1993). Additionally, the concept map tool was employed in a small 
number ol the intei views to elicit these students' conceptions about global warming: After the student 
responded to an interview question, the interviewer asked the student to concept map what the student 
believed to he tlu important parts of his/her response. At the close of these interviews, the researchers 
solicited th-' '■indents' perceptions of this process as a means to help them answer questions during the 
interview. The students' responses suggested the merits of the concept map. as process ard evolving 
product, to laeilitate the exlernali/alion of students' know ledge in the interview setting. Transcript 
excerpt-, ol students' perceptions follow, which speak to the supposition that the concept map may 
facilitate mctacognition lElavell. 1985k and elaboialion (Gagne. 1985) and explication by the student of 
their conceptual understanding: 

1 think it makes it easiei to understand what vou are saving. 

It | concept mapping parts of im answerl made it clcaivi so that 1 could understand what I 
thought about . When I think about things I just think about things one at a time This put it 
all together and I ean think about it more clearly. 
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As 1 was writing the things and I could see them there, it made inc think that there are aiso moie 

things that made, you know . . . just seeing them helped me think of more things. 

[T]hey [concept maps] give you ideas to the answers to the questions. 

The experience gained through and the findings (Rye, et ah, 1094) from the endeavors described 
above informed many aspects of the study set forth herein. These aspects included: (a) the development 
of interview protocols, (h) techniques for data collection and analysis that incorporated the concept map 
tool, and (c) the choice of cholorofluorocaibons (CFCs) as a subset of the domain— global atmospheric 
change tGAC)--in which to investigate students' conceptual understandings. Relative to the latter, many 
of the students who completed the STS-global warming instruction were found to hold alternative 
conceptions about the role of CFCs in global warming, including the idea that ozone layer depiction is a 
major cause of global warming (Rye ct a!., in press). The STS unit (Rubba. Wicsenmayer, Rye et ah, 
1995) that guided the instruction in this study had been revised to help students restructure such 
alternative conceptions. These revisions included expanding the scope of the unit to target other GAC 
issues, e.g.. ground level ozone pollution. Accordingly, instruction provided from this revised unit 
became known as “STS-GAC" instruction, and subsequently is referred to as such. 

Additionally, the unit revisions included an increased emphasis on the concept map. Several of 
the lessons incorporated concept maps and mapping activities. A self-instructional appendix on concept 
mapping also was included for science teachers, which focused on the “what, why. and how" of concept 
mapping and developing student competence in this learning strategy. 

The individual who served as the teacher-subject and reference “expert” in this study was a 
veteran science teacher and an exemplary participant in the Teacher Enhancement Project referred to 
above. She played a major role in the field testing and subsequent modification of the STS-global 
warming unit, and had considerable interest in the concept map as a science education tool. A volunteer 
sample of her students served as the student-subjects of this study. 

Purpose 

The primary purpose of this study was to investigate the effectiveness of two different types of 
post-instruction concept inters iews— one that did or one that did not embed a concept mapping process- 
us a means of eliciting students' post-instruction conceptual understandings about the nature of, source 
of. and problems caused by CFCs. In the “type of interview" that included concept mapping, the 
students wore asked to do more than just respond verbally to the interview questions: They also were 
asked to concept map what they believed to be the important parts of their verbal responses to the 
interview questions. Additionally, they were encouraged to examine (lie evolving concept map while 
considering their responses to the interview questions. 
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Several dependent variables ol conceptual understanding were defined (White & Gallstone. 

1992). Chief amongst these was the c.ileiion measure ol accordance (ACCORD), which tepresented the 
extent to which the student explicated during the interview the concepts: and concept relationships that 
comprised the ideal post-instructional understanding . This ideal understanding was based on the 
instruction provided by the classroom teacher and was set forth in a teacher-expert concept map. An 
important research question w;ts. Does tin interview that embeds concept mapping, compared to one that 
does not embed that process, increase significantly (p < .05) the extcrnalizalion of ACCORD? The 
corresponding null hypothesis was “Type of interview" does not predict ACCORD. 

I wo other criterion measures (VMiite & Gunstone. 1992) of students’ conceptual understanding 
examined in this study were as follows: (a) external related ness (EXTERN), as defined by all other 
GAC concepts set forth in concept maps that were placed at the beginning of each investigation lesson in 
the STS-GAC unit, and valid concept relationships between the teacher-expert and these other GAC 
concepts: and (b) interrelatedness (INTER), as defined by the mean number of relationships per concept 
(Stensvold & Wilson, 1990), where concepts were "tcachcr-expcrt" and "other GAC concepts” and 
relationships were those that comprised accordance and external relatcdness of understanding. Research 
questions and corresponding null hypotheses similar to those that targeted ACCORD were set forth for 
these additional measures of conceptual understanding, e.g.. Does "type of interview" predict EXTERN 
or INTER? 

A secondary purpose of this study was to answer the question, Is there any effect (including 
interaction) by the addition of' the independent variable, student to expert Pathfinder Network similarity 
index (PFNSI). on the prediction of ACCORD? Corresponding null hypotheses were, (a) The addition 
of PFNSI to a two-parameter regression model that includes "type of interview " docs not improve 
significantly (p < .05) the prediction of ACCORD; and (b) There is no statistically significant (p < .05) 
interaction between PFNSI and "type of interview" as it regards the prediction of ACCORD. 

This study also sought an answer for the question. What are the students’ perceptions ol the 
process instituted in the interviews as a means to “tell what they know" and what changes do students 
recommend to improve this process? Detailed findings relative to this question are presented elsewhere 
(Rye. 1995). However, set lbrlh herein are students' views of the concept map as a tool to "tell what I 
know." 



Methodology 

S ubj ect s and Research [Design 

Tins study was conducted in a junior high school that was located in a scmi-inban area of the 
northeastern I'nited Stales. Subjects were an eighth grade physical science teacher and students from 
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fi.'tir sections of her physical science course. Figure I illustrates the research design that guided this 
study, which is tied to the narrative that follows. 



Insert Figure I here 



Prior to the beginning of the study, the teacher introduced her physical science students to the 
concept tap tool and provided insti action from the foundation and awareness lessons < X f a ) in the STS- 

GAC unit. These lessons did not introduce the concept of CFC's. Subsequently, the teacher recruited 
student volunteers (S r ) to participate in this study, which surrounded the teaching of the investigation 

lessons from the STS-GAC unit. These lessons introduced and developed the concept of CFCs and their 
role in GAC. 

All of the student volunteers whose parents provided informed consent and release were 
scheduled as available and as time allowed for an initial concept interview (Oj). These interviews 

preceded the start of the STS-GAC investigation instruction, and henceforth are known as the pre- 
instruction concept interviews (PREIC1). A total of 38 students (approximately two-thirds female) 
completed a PREICI. which was conducted in accordance with a standardized open-ended interview 
protocol and chiefly for the purposes of removing the novelty to students of the interview setting. The 
PREICI did not embed a concept mapping process, but did include questions to elicit students' 
conceptual understandings about CFCs. Accordingly, these interviews generated transcripts that 
provided evidence of students' understandings about CFCs prior to STS-GAC unit instruction on CFCs. 
Additionally, the teacher verified that she had not provided any instruction on CFCs in the eighth grade 
physical science course up to that point, and to her knowledge, the students had not received such 
instruction in prior science courses. 

The week following the completion of the PREICI. the students began instruction from the 
investigation lessons (Xj) in the S TS-GAC unit. This instruction spanned a five week time period. 

Here, students had the opportunity to learn grade- appropriate science content that underlay the nature, 
causes, and potential consequences of global warming and ozone layer depletion, and as such, included 
instruction on CFC s. As part of this instruction, the teacher utilized concept maps to convey GAC 
content and assigned several concept mapping exorcises to the students. However, students were not 
assigned to develop concept maps in which CFCs were the focus or most superordinate (Novak & 
Ciowin. 1984) concept. 

Following the GAC investigation instruction, the students who were present for at least 75 H of 
this instruction and had completed a PREICI were randomly assigned (R u ) to two different "type of 
inters iew" groups (GA and GB) and "day of interview” tone of seven school days). Each of GA and GB 
consisted of 18 students. Seventeen of the students assigned to each group completed a second concept 
interview (CH or O.H, which yielded transcripts of students’ post-instruction understandings of CFCs. 
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Independent two- sample [-tests \eritied that these two groups did not differ significantly (p < .05) on the 
scaled scores from the grade 7 take of the reading verbal component (t = -.228, df = 5 1 , p = .82 1 ) or 
component total (t - .406, df= 31. p = .688) of the California Achievement Test (CTB McGraw-Hill/. 
I486). Additionally, the teacher certified that all students who completed either type of interview were 
competent in concept mapping. During the time span that the post-instruction inteviews were conducted, 
the teacher did not provide any instruction from the STS-GAC unit. 

The post-instruction conci pi interview (CD) completed by the students in GA was guided by a 
standardized open-ended interview' protocol that was very similar in structure and process to the 
PRFdCI: It did not embed a concept mapping process and was considered by the investigator to be 
conventional in nature. Henceforth, the interview completed by the students assigned to GA was labeled 
the post- instruction conventional concept intei view -POSTICCI. Like the PRFdCI, the POST ICC I 
posed questions to elicit students’ understandings about CFCs. However, it went beyond the PRE1C1 in 
that it posed questions to elicit students' perceptions of the interview as a means to “tell what I know," 

The post- instruct ion concept interview (O3) completed by the students assigned to GB differed 
in on ly on e respect from die POS FICCI: It was guided by a standardized open-ended interview protocol 
that was modified to embed a concept mapping process. The interview completed by the students 
assigned to GB is labeled the post-instruction modified concept interview— POSTIMCI. In addition to 
an interview transcript, the POSTIMCI yielded student constructed concept maps. 

After all of the post-instruction concept interviews were completed, a “paper and pencil” 
classroom activity w as conducted by the teacher with all of her physical science students. Here, students 
rated the relatedness of all possible pairings (11 = 28) of the eight concepts that had been deemed by the 
teacher as those concepts most central to an understanding about CFCs The concept rating activity data 
Irom the students participating in this study (Oq) were provided to the invcstigatoi, and utilized to 
generate student Pathfinder Networks and student to teacher Pathfindc. Network similarity indexes 
(Pl'NSI). 

Al ter the completion of the STS-GAC unit instruction from the investigation lessons, the teacher 
engaged in a knowledge engineering process with the investigator for the purposes of developing the 
teacher-experl referents (a teacher -expert concept map and Pathfinder Network) used in data analysis. 
This process is not reflected in the research design (Figure 1 ), and is presented under the data collection 
and analysis sections that follow. 



D ata Collection 

All data were collected by the first author of this paper. The tools used to collect data from the 
students were as follows: ia) the investigator us intciwicwci (Frlandson. Harris. Skipper. cN Allen, 
loo.l). (In standardized open-ended interview protocols (Patton. I‘»U()) to guide the PRHICI. POSTICCI. 
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and POS 1 IMCI (all interviews were tape recorded): and (c) for the POSTIMCI only, the concept map 
tool (Novak & Gowin, 1984) and map construction materials (a ceramic dry-erase hoard, markers, 
eraser, and blank sticky notes). Additional student data were provided to the investigator by the teacher: 
the scaled scores and national percentile rankings on select components (test total, reading total, reading 
vocabulary, and reading comprehension ) from the most recent (7th grade) take of the California 
Achievement Test; and the results from the classroom paper and pencil activity w here students rated the 
relatedness of pairs of central GAC concepts. 

Data were collected from the teacher lor the purposes of obtaining teacher-export referents (a 
teacher-expert concept map and Pathfinder Network) to be used in the analysis of student data. These 
data were collected through two post-instruction interviews (guided by the protocol used to conduct the 
POSTIMCI). a concept map and map construction materials, and a paper and pencil exercise that 
required rating the relatedness of pairs of central GAC concepts. 

Investigato r 

The investigator's experience as an interviewer in and as an analyst of data from research that 
sought tti investigate students' conceptual understandings, including the employment of the concept map 
tool in such research, was set forth within the background section of this paper. The investigator in this 
study possessed a sophisticated understanding of GAC relative to the science content that comprised the 
S TS GAC instruction for the students in this study. In reporting their results from a longitudinal study 
that incorporated interviews and concept maps to investigate students' science understandings. Novak & 
Musonda 1 . 1991 ) point to the importance of the interviewer's knowledge base in the content domain in 
which students' understandings were examined: "We found that it was essential for interviewers to have 
a sound grasp of the science concepts involved in the interview. . . " (p. 124). A sophisticated 
knowledge of the content domain, relative to what the students' being interviewed are likely to possess, 
is critical to the interviewer's formulation and issuance of declarations of complexity (Dana. Kelsay. 
Thomas. Tippins. 1992). Such knowledge also is important to data analysis, where the investigator’s 
knowledge is called upon to make decisions about the validity of certain concept relationships explicated 
in the student interview transcripts and concept maps. 

Stu dent Interview s 

All interviews were conducted b) the investigator in a private setting in the junior high school 
during normal school hours, were tape recoided. transcribed, and verified against the tape. The PRHICI 
were conducted over a threc-day period immediately prior to the start of the STS -GAC investigation 
instruction atui ranged in length from 15 to 25 minutes. The post-instruction interviews (POSTICC1 and 
POSTIMCI ) began approximately one and one-half weeks after the completion of the investigation 
instruction and were conducted ovei eight sequential school days. The POSTICC1 ranged in length from 




1 7.5 to 39 minutes and the POSTIMCJ. from 28 to 50 minutes. Approximate!) equal tuiinhers of each 
type of interview were conducted on each day 

The complete protocols employed to conduct the PREICl, POSTIC’CI. and POS n.YK’l arc 
provided elsewhere (Rye, 1995). These interview protocols conformed to struct oral and procedural 
specifications that have been set forth by recognized authorities in the area of eliciting students' 
understandings via (lie interview tool (Novak & Gowin, 1984; Osborne & Freyherg. I 9K5; White A 
Gttnstone. 1992). For example, the interview questions posed to externalize students' understandings 
were open-ended, non-leading, and sequenced from somewhat broad to more .specific. 

1 hose inters iew protocols evolved Irom those that were refined over a two-vear period b\ the 
investigate) s in the teacher enhancement project research, described previouslv in the background 
section of this paper (Rye ei al.. 1994. in progress). Additionally, a separate study wa • conducted to 
pilot-test these protocols. This study is reported elsewhere (Rye, 1995), and resulted in minimal changes 
to the interview protocols. 

1 he PRF.1CI. POS I ICC I and P()S f I, MCI each were designed to include the follow mg 
components: (a) a verbal explanation by the interviewer of the nature, confidentiality, purpose, and 
processes involved in the interview, including that the interview would pose questions about CTCs: dn 
up to six minutes of quiet time (Heinze-Fry & Novak, 1990) for the student to think about CFC's and to 
jiit down thoughts and ideas (as single words, phrases, sentences, or diagrams) that they believed ueic 
important, for their future reference while answering the upcoming interview questions about CF(\. .ci 
questions (to be asked by the interviewer) that attempted to elicit students' conceptual understandings 
about the nature of, sources of, and problems caused by CFCs (Table 1 ): and (d) the piovision bv the 
interviewer of refloetivo/verifieution statements in response to the student's answers to the initial asking 
and one reasking of each question in Table 1 . 



Insert Table 1 here 



The issuance by the interviewer of refleetive/vxrification statements (Danaet al . 1 992: Spradelv 
1979), declarations ol perplexity (Danaet al.. 1992), and follow-up probes, in response to students' 
verbal answers to interview questions, were standardized across the POSTICt ’I and POR'ITVICI 
protocols. The frequency with which the interviewer repeated interview questions and provided 
refleeti ve/verification statements also was standardized across protocols. Probes and responses other 
than those set lorth in the protocols were avoided by the interviewer. The investigators acknowledge 
that selective ptobing of students' responses by the interviewer, in order to investigate more deeply an 
'interesting remark" made by a student, is a recommended interview practice (Novak A Govvm. |9S4. 
Osborne A Freyherg, 1985: White & Ciun.stone, 1992). However, in this study, such selective ptnbmg 
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would have been a serious limitation, and invalidated comparisons between students completing the 
POSTICC1 and POSTIMCI. 

The investigator purposefully designed the protocols that guided the post-instruction concept 
intervievvs-POSTICCl and POSTIMCI— to he the same in structure, content, and process, with one 
principal exception: The POSTIMCI embedded a concept mapping process. The embedding of the 
concept mapping process translated to three aspects of the POSTIMCI protocol. aU surrounding the 
evolving concept map. that were not a part of the POST1CCI proto. ol. These aspects were that the 
POSTIMCI (a) involved the student in constructing a concept map that set forth (as concei and 
concept relationships) what he/she believed were important parts of his or her answers to certain 
interview questions, tb) included dialogue between the interviewer and student relevant to the concept 
mapping process and what was being mapped, and (e) explicated an evolving visual (the student's 
concept map) for the student to view and reflect upon as the interview ptogressed. 

Detailed descriptions are provided elsewhere (Rye, 1995) of the procedures employed to conduct 
the POSTIMCI. which began wi*h orienting the student to the interview-based concept mapping process. 
At the close of this orientation, t.ie interviewer told the student: 

Since this interview is about CFCs, I will write this (CFCs) concept down on a blank sticky note 
and I would like you to use it in your concept map. You can use it when you start your map or 
you can map it in later. This is the only term that I will ask you to use: You will decide all of 
the other concepts that go in your map. 



The sticky note of CFCs was then placed on the side of the dry -erase board that was to be used for the 
concept mapping. »Vhat follows is a summary of the procedures employed, beginning with the first 
interview question (Table ! ). to integrate the concept map construction process with the posing of those 
questions, the interviewee’s verbal responses, and the interviewer’s reflective statements: 

1. The first interview question was posed. After the student responded, the interviewer issued a 
reflective response followed by deliberate silence. If the student came forth with additional 
verbal responses, the interviewer provided corresponding reflective statements. Then, the 
interviewer asked the student to think about his or her past rcsponse(s) and state: "What are a 
couple of terms {concepts; in what you just said, that you consider to he most important?" 

2. The student verbalized the term or terms and the interviewer printed them on separate sticky 
notes. The sticky notes containing the terms we r e placed beside the sticky of CFCs, resulting 
in two to three terms available to start the mapping. 

3. The interviewer asked the student to start constructing the concept map by (a) selecting the 
two terms on stickies that he or she wanted to use to begin the map anil ib) mapping in the 
connection, while thinking about his or her past response to "What are CFCs?" 

4. The interv icwer told the sli’-’ent that the student could talk about whai he or she was mapping 
in. if he or she so desired, because the interviewer was "interested." 
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5. I he intei viewer reminded the student of the following, ta) to not be concerned with how neat 
the map looked, (b) to not spend too much time in deciding how to map the tenns-that the 
interviewer was more interested in what he or she had to say than in the map--, and (c) that he 
or she could change the map at any time. 

6. If there was another concept remaining on the side of the board, the interviewer gave the 
studen’ ‘he option of mapping it in, too. 

7. The interviewer posed the interview question a second time, which was followed by 

additional reflective statements by the interviewer and mapping if the student had any further 
• response(s). 

8. If the student continued to provide verbal responses to a third and fourth asking of the 
question, these responses were not reflected or mapped-thc interview time block was 
insufficient to allow for this. 

9. The above process (1-8) was repeated through question 31;. The interviewer repeatedly 

reminded the student that he or she should examine, the map while thinking about how to 
respond to askings and re-askings of each of these interview questions. 

1 0. After the student had no more to say in response to questions 3 through 3b. the student was 
asked if he or she was satisfied with the map, or if final changes were needed. The student 
was told to examine the notes he or she had jotted down during quiet time in order to reach 
this decision. The student subsequently made any changes she or he deemed necessary, and 
the concept mapping component of the interview came to a close. This process allowed for 
student verification of the concept map. 

1 1. The concept map remained out on the table for the remainder of the interview, and the 
student was encouraged to examine the map when considering what responses she or he 
might make to remaining interview questions (Table 1. questions 4 and 5). 



The POSriCCI and POST 1 MCI protocols contained questions beyond (and posed after) those set 
forth in Table 1. These additional questions were for the purposes of (a) assessing the students' ability 
to transfer their understanding about CFCs to scenarios about atmospheric levels of CFCs and ozone, 
and (b) eliciting students' "views and opinions" on the interview as a means to “tell what I know." The 
transfer questions and related findings are presented elsewhere (Rye, 1995). The “views and opinions” 
questions arc presented in Tabic 2. Because the POSTIMCl embedded a concept mapping process, the 
set of questions to elicit these students views and opinions was expanded to include those that honed in 
on the concept map tool ( Table 2. “Additional Questions for Students Completing POS TIMCl"). In 
responding to these “views and opinions” questions, the interviewer encouraged the students to be frank 
--to not say something just to make the interviewer happy 



Insert Table 2 here 
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Te ach c i Interviews 



After the teacher finished providing the students with the STS-GAC investigation instruction, 
two interviews were conducted with the teacher, which embedded processes similar to those described in 
the knowledge engineering literature by Gordon and colleagues ( 1989; 1991 ; 1993) for extracting an 
expert's knowledge of certain content in a given domain. In this study, these processes were applied to 
construct an expert referent— a teacher-expert concept map- -for use in analyzing a portion of the student 
interview data, and for determining central concepts for Pathfinder analysis. These teacher interviews 
are described in detail elsewhere (Rye. 1995) and a summary is presented below. 

The POSTIMCI protocol was employed as a guide for the first of these two interviews with the 
teacher, which resulted in a draft of the teacher-expert concept map. This map explicated what the 
teacher believed to be the ideal post-instructional understanding that students should hold, relative to the 
interview questions (Table 1 ) posed. The interviewer made clear that his intention was run to assess the 
teacher's knowledge or elicit her personal understanding, but rather to determine what she desired, 
ideally, the students to know, based on the STS-GAC instruction she provided. 

A copy of the draft teacher-expert concept map. along with a copy of the interview transcript and 
the interview questions (Table 1 ). were provided to the teacher to reflect on during an 1 1 day period that 
lapsed between the first and second interview. During this time, the teacher was asked to note changes 
(including the addition of cross-links) that she believed were necessary in order for the expert map to 
reflect better the ideal understanding. The interviewer also studied the transcript and expert map. and 
identified sections in the transcript and map that warranted probing at the second interview. 

The second interview principally was for the purposes of producing a final copy ( Figure 2) of 
tins teacher-expert map. Accordingly, the draft map was modified and verified by the teacher during the 
interview. This final map was employed later by the investigator as an expert referent in constructing 
templates of concepts and concept relationships, which were applied in the analysis of students' verbal 
responses (anti concept maps in the case of the POSTIMCI) to the questions in Table 1 After the 
teacher verified the expert map during this second interview, she subsequently identified the 8 concepts 
that she deemed “most central" to the understanding set forth by the map. These 8 concepts were as 
follows: global warming, greenhouse gas, I'V light, depletion, man made, ozone layer, coolants, and 
eholorofluorocarbons. These concepts formed the basis ol' the concept relatedness ratings utilized to 
generate and compare Pathfinder Networks, as described below. 



R e k tted nes s Ratines o f Co nce pts 

During the planning stages of this study, the investigator explained to the teacher his interest in 
i ollectmg post-instruction data from student subjects and the teacher that could he used for Pathfinder 



Insert Figure 2 here 
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analysis. Specifically, this data would result from an activity where the students and teacher rated ( 1 to 
9 Likcrt-lype scale) the relatedness ( 1 = none or very w eak and v = very strong) of all possible pairings 
ol the 8 concepts deemed as "most central" by the teacher in the teacher-expert concept map (Figure 2). 
The algorithm [n(n- 1 ) / „| (Acton, Johnson, & Goldsmith, 1 994) tor computing the maximum number of 
concept pairs from some number (n) of individual concepts results in 28 concept pairs, when n = 8. 

The teacher was interested in having all of her physical science students complete this concept 
rating task, and later, use their ratings in a debate-type classroom activity about the relatedness of these 
concepts. She subsequently agreed to provide to the investigator the ratings of only those students who 
were participating in the study. Making the task an established part of the science course potentially 
increased the reliability of the results of the concept relatcdness ratings. That is. students likely viewed 
the activity as being a part ot their science course as opposed to a task to complete solely for some 
research study. 

The investigator derived the concept rating activity and related instructions based on the 
procedures set forth by others (Acton et al.. 1994; Jonassen, 1993: Sehvancveldt, 1990) to collect such 
data lor the purposes ol producing Pathfinder Networks and obtaining the similarity index of student to 
expert Pathfinder Networks (PFNS1) Specifically, the ordered list of the 28 concept pairs w'as derived 
by. (a) generating a random sequence ol the eight central concepts, and (b) entering this random 
sequence of concepts into the Know ledge Network Organizing Tool (KNOT rM , Interlink Inc.. 1993) 
computer program, which in turn generated the ordered list of the 28 concept pairs. The 
activity/directions were verified by the teacher as something that students readily could understand. 
Additional expianations/instructions were provided orally by the teacher to help insure that student.-, 
would take the exercise seriously, and that useable data would result. These instructions included the 
follow ing; (a) the purpose ot the activity (i.e. would be used later in a debate); (b) a practice exercise 
where students rated the relatedness ot concepts comprising I ami liar concept pairs (e.g., boat-water, 
boat-land) unndaied to the STS-GAC instruction; (c) taking no more than 15 to 20 seconds to make a 
decision about the relatcdness of any concept pair; (d) to circle "5" on the rating scale if they had no idea 
of the degree of relatedness between the concepts comprising a given concept pair (Acton ct al., 1994). 

Thirty-three of the 34 students who participated in the post-instruction interviews subsequently 
completed the activity. The investigator also administered this activity to the teacher. The teacher 
followed the same directions as did the students in completing the activity. KNOT ,M was used to 
produce Pathfinder Networks from each student s and the teacher’s ratings, and subsequently to compute 
the similarity index for the teacher’s (expert's) and each student's Pathfinder Network (PFNSI). The 
derivation and use ol the PFNSI are described more completely within the data analysis section. 
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Da l a An alysis 



The POSTICC1 and POSTIMCI data (interview transcript from all student and concept maps 
lrom POSTIMCI) wcic assessed against lefereni templates to determine the degree to which each 
student achieved the criterion measures of understanding: ACCORD, HXTPRN. and INTER. 

Speed ically the content of each POSTICOI and POSTIMCI transcript through student responses to 
interview question 5 ('fable 1 ) was unili/ed via a case analysis pioeess (Patton, 1990) to determine the 
presence of’ the concepts and relationships that comprised the measures of ACCORD and EXTERN, and 
these concepts and relationships were marked/entered on the data analysis templates (described below t. 
fan* students completing the POS TIMCI. the concept maps emergent from the interview s also were 
examined via a case analysis process to identify the presence of any additional teacher-expert concepts 
and relationships that were not explicated by the student ver bal 1\ , i.e.. were not contained in the 
interview transcript. The data analy sis templates also were marked to reflect the explication of these 
additional concepts and telalionships. The sums of these concepts and relationships marked on the 
templates were used u* dense quantities for the extent to which the criterion measures of ACCORD and 
EXTERN were present. 

flic measure of INTER -a ratio of the concept relationships to the concepts explicated -was 
derived from the concepts and relationships marked on the ACCORD and EXTERN templates. 
Specifically, the total number of icachei expert relationships plus relationships between teacher-expert 
and externally i elated CiAC concepts (described below) explicated by the student was divided by the 
total number of teacher-expert plus externally related GAC concepts explicated by the student. This 
quotient constituted tlieintio INTER. 

Bi\ ariatc regression analyses (Judd A McClelland, 19X9) of data, perfumed using MYSTAT 1 M 
STA R 199 1 ), were used to make decisions to retain or reject the null hypotheses about “type of 
interview" predicting each of ACCORD, EXTERN, and INTER. Multiple regression analyses wore 
performed to determine it' students* P1NSI improved the prediction of ACCORD, or interacted with 
"type of interview" in predicting ACCORD. Frequency and inductive analyses (Patton, 1990) were 
ctnploved to analv/e student transcripts to determine their perceptions of the interview as a means to 
"tell what they knew" i Table 2). I hc fiequeney analysis was stmilai to a classical content or protocol 
analysis and the inductive analysis look the form of a G rounded Theory approach, where the transcript 
data was unm/cd. coded and labeled (Piugcon et al., 1991 ). An attempt was made to combine thiougli 
logical anaivsK i Fallon. I990i emergent categories (Marshall A Rossman. 1988). 



I empfajcs 

The templates Or assessing ACCORD. EXTERN, and IN'IER were derived prim to the analysis 
of any o| the post instruUional interview data. 'The teacher-expert concept map (Figure 2) served as the 
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re lei cut lor developing the ACCOR!) templates: and the concepts in the teacher-expert concept map and 
in the concept maps at the beginning of each STS CiAO investigation lesson vveie the referents for 
constructing the EXTERN templates. 

A CCORD T emplates 

One template for ACCORD listed the 40 concepts from the teacher-expert map and the second 
template listed the 45 concept relationships from the map. The analysis of concept relationships as a 
measure of student understanding from interview transcripts has been described as "concept 
propositional analysis' (Novak & Govvin. 1084; Posner & Gert/og. 1082) and referred to by others as 
the use ol "expert-derived" or "critical” propositions'll.. omask et a!., 1003; Wallace & Mint/es. 1090t or 
uist propositions relative to the knowledge an individual holds about concepts (White & Gunstone). The 
teacher-expert relationships listed on the concept relationships template included propositions explieillv 
illustrated on the expert map (e.g . ('PCs are used in coolants), explicitly illustrated cross-links (e.g.. 
CbC substitutes may also be greenhouse gas), and propositions that were implicit in the understanding 
set lortli by the expert map. The process whereby implicit propositions were determined, and limitations 
associated with this process, are described in detail elsewhere (Rye, 1995) and summarized below. 

Implicit propositions were instances of two concepts in a progressively differentiated subsection 
of the expert concept map (Figure 2) hierarchy dial were not directly linked, yet could be related in a 
\ alid and relevant matinei through linking words that existed in that same segment of the concept 
hierarchs . Examples of implicit propositions were "CFCx arc used in air conditioners,” "CFCs lead to 
an increase in global tcmperature/global wanning. " and "global warming alters species." The 
investigator's decision as to what constituted the wording for and realm of implicit propositions was 
influenced by his immersion in the knew ledge engined ing process w ith the teacher to produce the 
expert concept map. which included a study of the transcripts that resulted from the interviews with the 
teacher. Additionally, the investigator was influenced by his in-depth knowledge of the STS-GAC unit 
used as the instruction in this study. Accordingly, a fair cross-validation process (for the determination 
of implicit relationships) would require other investigators to be equally immersed, involved, and 
knowledgeable. 1 he fact that the classroom teacher did construct and validate through a reflection 
process the concept map from which the investigator extracted the relationships did constitute to some 
degree a validity check. 

Concepts and concept relationships explicated by the student that were, in the judgment of the 
investigator, very similar in meaning to those on the templates were counted as semantic equivalents. 
Such semantic cquliv alcncv included relationships that were asymmetric equivalents, e.g.. “Humans 
make C FCs" is the a>v mmetric equivalent of “CFCs aie Human-made." Furthermore, the decision to 
lodge a concept relationship explicated In the student as the semantic equivalent of a teacher-expert 
concept relationship was made vv it bin the hroadei context of what the student explicated prior and 



subsequent to that concept relationship II mil h surrounding explications aie not considered in the 
determination of i elation ship equivalency . then the data analysis is characterized as being an isolated 
f actual approach as opposed to one that is contextualized in the students' broader understanding. 

The determination of propositions that are implicit in an expert concept map. and the use of those 
propositions in analyzing understandings captured by interview transcripts, goes beyond what has been 
reported elsewhere. Such an analysis accounts for valid and relevant (to the expert map) understandings 
that may be omitted In referring only to explicitly stated expert or critical propositions (Rye et ah. 

1994). The taking into account of implicit propositions, similar semantic meanings, and asymmetrically 
worded relationships foi assessment purposes is congruent with the constructivist paradigm that guides 
this study -- it recognize*, that knowledge is indi\ iduully constiueted and to varying degrees idiosyncratic; 
and that expeit concept maps should not he "straightjaekcts" (Lomask et ah. 1993. p. 5) for assessment. 

E XTERN Templates 

The STS-GAC unit investigation lessons that comprised the instruction for this study each 
contained a concept map. \v hich set foi th impoi tan t understandings about the lesson. I here were 1 4 1 
different concepts in these investigation lesion concept maps. Practically all of the concepts in the 
teacher-expert map also were in these lesson concept maps. This total of 141 OAC concepts was reduced 
lo N7 concepts by eliminating the teacher-expert (i.e., the ACCORD) concepts as well as several 
additional investigation lesson concepts that were the semantic equivalents of teacher-expert concepts. 
These X7 investigation lesson concepts weie the externally related GAC concepts, and comprised the 
concepts template for h’YI CRN. A second template was derived foi the rounding of all valid (in the 
judgment of the ime^tigatoi ) iclationships between the EXTERN and ACCORD concepts 

After the investigator completed the anulv sis of student transcript data against all of the 
templates, he conferenced w ith a colleague -the second author of this paper, who also was very familiar 
with the STS-GAC unit* on a small number of problems that arose during analysis. This colleague 
helped the mveMignioi J o establish piocedute to resolve such problems, and the investigator applied such 
procedure in a second examination of the data. A detailed description of these problems, along with the 
corre- ending decisions applied to the re-examination of the data, are piovided elsewhere (Rye, 1995) 

Pi' NS J 

The concept iclateilness ratings of the 2S concept pairs, completed by each student and the 
teacher, were entered b\ the investigator into KNOT ,M . KNOT IM transformed these ratings, utilizing 
* I i c Paihlnuier sealing algorithm, into a Pathfinder Netwoik foi each student and the teacher. The 
teacher s and example students' Palhfiiulei Networks are illustrated (as Figures 3-5) w ithin the findings 
v.viioi: o! this papei. The nodes in these iemlung Pathfinder Networks are the eight concepts deemed 
in the teacher lo he tin we most tcnlial in the understanding set hath in the teacher-expert concept map. 
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I he sc concepts arc at ranged spatially accouling to the relatedncss ratings (semantic proximities) of the 
concepts comprising the concept pairs. 1 wo nodes ate linked il the sealing algorithms determine that 
this link explicates the mini'*- mi length path between the two nodes. 

KNOT !M subsequently was employed to compare each student's to the teacher's net to generate 
the Pathfinder Network similarity index fITNSh for each student. PFNSI is a quantitative measure 
ranging from 0 (nets have no similarity) to 1 (nets are identical). 

Findings and Discussion 

Cri ten on Mea su r es of Co nce ptual L nderst anding 

I able 3 sets forth dcseripti\e statistics on the degree to which s udents who completed each i\ pc 
ot post- instruction eoriccpt interview (POSTICCI or POSTIMC1) explicated the criterion variable 
measures of 'under standing* ACCORD, which is the sum of teacher-expert concepts (ACCORDC) and 
ielationships ( ACCORDRi, HXTHRN, which is the sum of externally related GAC concepts 
' FA' 1 FRNC) and relationships (FX4 FRNRk and INTHR, which is the ratio of teacher-expert plus 
externally related relationships to teacher- expert plus externally related concepts |i.e.. (ACCORDR + 
FXTHRNR ) + (ACCORDTC + F;XTKRNC)|. 



Insert Table 3 here 



Bivariate regression analysis wheie ACCORD was regressed on the independent variable type 
of interview ” (TYPLITVj tailed t ' reveal TYPK1TV as a statistically significant predu toi i b ~ -3.059; p_ 
~ .46 7 , R- =■ 017). i e.. the interview that embedded a concept mapping process did not result in a 
statistically significant inciease in explication of the teacher-expert concepts plus ielationships that 
comprised accordance nt\ oneeptual understanding, f YPF4TV also- failed to predict the explication of 
either ACCORDC (J2 = .310) or ACCORDR (p= .679). Actually, the data in each ease appeared to 
favor shghilv the type oi interview that did mil embed a concept mapping process, as illustrated b\ 
comparing the accordance mean values foi POSTK'CI and POSTIMCI ('Fable 3). 

Bivariate regtession analv.sis where hXTHRN was regressed on TYPLdTV also failed to reveal 
l'YPKI FY as it statistical!) significant predictor 'b = 1.294: p - .542; R~ - .012) The results 
paralleled Boost* obtained for accordance of unde* standing: ( iU TYPPITV also failed to predict 
explication of either I iXTFKNC (£ = .544 ) or liXTFRNR (p ~ .565 }, and (b) the data in each case 
appeared to favor slightly the tv pc of interview that did not embed a concept mapping process. 

Rival late regression analysis while INTHR was regicssed on TYPFITV failed to reveal 
IYPKITY as a statistically significant predictor lb = .024; p - .669. R- ^ .0()6|, However, the data did 
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not appear to favor the P( )S l K C l group in lilts case. Potent ial explanations tor the non- significant 
1 imlings ot TYPLI 1 V us a predictor of the measures ut' conceptual tmdeistandine are discussed later 
under “Students' Perceptions of Inlcr\ icw A 

Pa l h f_ i n dje r _N e t w o rk_ S uj] dan tv; _ Index m J Ted ic t i i ig_ A.c e o rdun l e 

Ihe multiple rcgicssion models employed to detetmine if students' Pathfinder Network 
similarity index (PFNSI) impnned significant 1\ tp< .05) the prediction of ACCORD, or interacted with 
TYPHITY in predicting ACCORD, were built upon the bivariate (two-parameter) regression model 
utilized to investigate TYPHITY as a piedietor of ACCORD. Desciiptive statistics on PFNSI loi 
'indents' completing the PC )S I IC'( 'I and POSTIMCI were, respectively, as follows: M = .454 (range of 
250-.7X6). S . D . ~ . 1 30. M - .4-44 ( range of . 1 58 -.733 ). S.D. = .125. Tlie group means weie almost 
identical: results of the independent t-tcsl u = .226, p = .S23) show ed they did not differ significant !\ . 

The teachers Pathfinder Nctwoik (i.e.. the expert referent) against which all students' Pathfinder 
Netwoihs were compared to generate each students' PFNSI is illustrated below as Figure 3. Figures 4 
i.w\ 5 illustrate, respectively, the Pathfinder Networks of the two students who had the highest (.786) and 
lowest (.158^ PFNSI \ allies, A \isual inspection of the teacher-expert Pathfinder Network (Figure 3) 
and Figure 4 suggests a high degree of global similarity (Goldsmith el al., 1991 ) exists betw een the 
cognitive structures (iclalive to the concepts shown) of the teacher and the student with the highest 
PFNSI. These two nets have 1 I links in common and the location of most of the concept nodes (c.g.. 
global warming and man made) is very similar. Kach net locates chlorofluorocarbons (CTCs) most 
centrally . and intei links it mo>t extensively w iih neighboring concept nodes. 



Insert Figures 3-5 here 



CoiisidciabU. dissimtlai ily appears to exist m cognitive stiueture between the teacher and the 
student w ho had the lowest PFNSI ( Figure S k with only 3 links common to both nets. This student's net 
also luis fewer total links than either of the nets sot forth in Figures 3 and 4. This illustrates that the 
Pathfinder algorithm, winch transformed this student’s concept ratings into a structural representation, 
relumed fcvvci links than was the case foi (he teacher and student icpiesented by Figuic 4. As such. (Ins 
student's (Figure 5) cognitive strueuue relative to the concepts shown appears to be less interrelated than 
that of the teacher or othci student. Addu innally . unlike the nets in Figures 3 and 4, this student s net 
;nlc! links io the greatest degree and locales most centiullv the eoikept node "man-made, " with CFCs 
occupy mg a space < m ihe pei iphery . This student also had the lowest ACCOR D value of all students 
w ti*» completed the concept inicivicws. Oomci.sclv, the student with the highest PFNSI was 
v nmadciahlv above the mean value loi ACCORD. 
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The results fiom multiple regression analysis, where ACCORD was regressed on TYPH1TV and 
PFNSI. showed that PPNSI improved significantly tb = 42.2X6; p = .003: ~ 266) the prediction of 

ACCORD over a two-parameter model that contained TYPITfV as the sole predictor. The findings 
fiom multiple regression analysis that tested for the interaction '‘TYPHIFV x PFNST in predieting 
ACCORD revealed that it was not worthwhile (b = 7.004; p ~ ,X21 ; R~ - .002) adding this predietoi to 
the regression model. 

During data analysis, the investigatoi became interested in whether PFNS1 also would improve 
significantly (p < .05) the prediction of students' scaled scores (verbal and total ) on their most recent 
take of the California Achievement lests (C.VI ). Post hoc multiple regression analyses showed that 
PPNSI did improve significant!} the piediciion of both the verbal (p = 001 ) and total (p = .003) CAT 
scores. These post hoe tmdings coupled with the fact that PFNSI was a strong predictor of ACCORD 
lend additional support to the utility of Pathfinder Network structural representations of knowledge, and 
most specifically the \ alidity ol’PTNSI, as measures of school learning (Jonassen et al.. 1093). 

Goldsmith ei uh ( 199 1 ) stale that Abe validation of a structural assessment should occur within 
the context of acquiring general classroom knowledge" (p. 94). They also contend that "the validitv of 
the structural approach is better evaluated if the concepts used to derive the struct ure are only a sample 
ol t he set of concepts on which peiformance is assessed’’ ( p. 94). and state further that such performance 
assessment should reflect the breadth of understandings and skills that are desired relative to the content 
area being assessed. In this study. PPNSI was derived from the students' concept relaledness ratings of a 
critical subset of eight ‘'central’' concepts, which were embedded in the measure of accordance 
(ACCORD) ol conceptual understanding. Here. PFNSI correlated strongly (r= ,515, p = .003) to 
ACCORD Goldsmith et al. take this eorrelanon as the measure of predictive validity of the structural 
assessment. 

Given the algorithm | n( n- h / 2. w heie u equals the numbei of concept] for determining the 
number of concept pairs to he rated to generate the measure of PFNSI, it is ideal if predictive validity for 
PFNSI can be established based on a relatively small number of representative concepts (Goldsmith el 
al.. 1991 ). 'This scents especially important in the case of working with younger learners, where time on 
task is more of an issue. In this study , a relatively high level (r = .51 5) of predictive validitv was 
established for PFNSI from the ratings of just 8 concepts n.e., 2X concept pairs), which were deemed by 
the classroom teacher as being most central in an expert concept map constructed by that teacher. By 
comparison. Goldsmith et al. estimated that between 15 and 20 central concepts would be nceessarv to 
achieve tins level ir~ .515) of predictive validity for the measure of Closeness on course grades in a 
college-level research psychology course. Here. subjects rated 30 concepts (i.e. 4X0 concept pairs) 
deemed most central to the domain by the instructor, and the resulting predictive validity for Closeness 

was T4 
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Goldsmith et ul do not specify what pioccdiire was utilized In the psychology course instructor 
In extract those 30 "centiar concepts Imm the huger body ol domain-specific knowledge. Perhaps the 
const! action and vci ification In the teachei of an expert concept map in this studs facilitated the 
subsequent selection by the tcachei of those concepts that truly were mo st central to the domain ((*F(\ 
and their role in (iACi, and at cordingly . for which the concept lelntedness ratings woukl ultimately 
yield a PI-.VSI with substantial predictive power. On the other hand, perhaps the high predictive validity 
ol Pf’NSI for ACCORD was due to the fact that both predictor iPFNSI) and criterion ( ACCORD) were 
derived through comparison to teacher-bused siiuctural referents -lhe teacher's Pathfinder Network and 
“expert*' concept map. respectively. 



Student^ Pejcepfio.ns of Interview 

fhe findings of students’ peiceptions of the inteiv ievv process as a means to tell w hat they knew . 
as elicited through the questions in Table 2. are presented in detail elsewhere (Rye, 1995). The findings 
set forth below generally are limited to POSTIMCI students* vievv.s of concept mapping as an interview 
component. and are triangulated with the icsiilts fiom regression analyses in the conclusions section. 
However, also important to the coik lusions of this study was the value that all students ascribed to the 
"quiet time*' for thinking and jotting down thoughts about CFCs prior to the considering t he interview 
questions. This component was perceived as helpful h\ the most (X2 l 7 ) POSTICCI students (n = 17), 
and b\ almost bOA of POSTIMCI students tn ~ 17). Students supported this perception with a variety 
of reasons, ranging from using this think time met acogni lively (to brainstorm and organize thoughts), io 
using i he notes as an aid while consideiing specific interview questions. Prior to reporting the findings 
nf POSTIMCI students* perceptions ot the concept map as an interview component, brief attention is 
given to the structure and vaiiety ol concept maps that emerged from these interviews. 

Maps Fmc rge nt f rom P( XS'llMCl 

The 17 concept maps prepared by the students during theii POSTIMCI varied somewhat in 
struettue and even more so in the number oi concepts and concept relationships The mean number of 
r< incepts per map w as about 1 0. with a range of Mo 17 The number of concept relationships ranged 
trom 2 to 1 N, with a mean of 9. Pew of the concept relationships in any map were explicated as cross- 
link Seven ot the maps contained r.o cross links and the remaining 10 maps explicated a mean of 2 
.Toss links 

Figures 0 tluoiiph X ate concept maps piepaied by three students during the POSTIMCI. and 
have been selected to illustrate the variety of emergent maps in terms of framework and organization, 
blanching or progressive differentiation of concepts, and cross linking or integrative reconciliation of 
i rm epts * Novak. I 99 ' i Figure 9 illustrates one of the least complex of all concept maps produced by 
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the student v Although il is hierarchical!) i ranted . it is basically linear with minimal branching of 
concepts and no cross-links. Approximately one-half of the maps were chiefly hierarchical in structure, 
and most explicated more hoii/oulul breadth than the map m Figure 6. 



Insert Figure 6 here 



Figure 7 illustrates one of the more complex of all maps produced by the students It is 
hicrarchicalK I rained and represents the extreme in terms of articulating clearly the levels in the concept 
hierarchy. In many of the maps that were liieiaiehicallv framed, it was difficult to discern levels in the 
concept hierarchy . Howc\cr, tins may ha\e been due in part to the interviewers remarks to the students 
to "not worry about being neat." The interviewer did not want students to become preoccupied with 
neatness, as this would detract from diinking and talking about CFCY 



Insert Figure 7 here 



The map illustrated in Figure X is a hvbrid (Weslet ah, IWi ) of web and hicraiehical structures, 
as w as tlie case in scvctal of The other maps . It explicates the extreme in terms of cross-links, but clcarlv 
all of the links with arum heads are not true cross-links. The use of arrowheads on linking lines is a 
sonventio. to distinguish cross links from other concept relationships (Novak & Gowin, 10X4), and the 
students in tins sttn!_\ wue appraised of this convention. This niap also was interesting because CFCs 
were not explicated as the focus concept in the map -that concept appears to be "strato. o/one. * In the 
mainrity of maps produced bv the students. ('FC\ was the focus or superordinate. 



Insert Figure X here 

Pn.S JJ C;M1_S Likie.nl s * I\m cep t tons of Mapp it ig 

The first iiitoivievv question (Table 2) to elicit students* perceptions was wnuled in a verv open- 
ended manner, the intent being to avoid cueing students to think about any specific component. In 
i espouse, ovet 40 r t o\ the students stated that the concept map was helpful in some wav One of these 
students. 1 ucillw speaks to the "graphic oigani/er*' tWandersec. I WO) feature of a concept map: 



Lucilie: And it was good to do a concept map. too. because il kind of helped you sort xoui 

tlioughls out . . >n\ou ran kind of see what sou aie thinking 
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Lucille's continent suggests that she employed tin map tuetacognitively- as a "strategic action ol the 
reasoner" (La Ion •k I .inn. i l )XN)--u. help orgum/i her thoughts. Jane infers that the concept map pjoved 
a mle in 'Apicad-of-ucliv ulion'' ((Jayne. IWO). leading to furthei recall of her knowledge. 

Jane: [WJhen I would sec this one thing, it would lead to another thing that I would, maybe, 

didn't think about before. 

khiesiion* 2a through 2d (Table 2) attempted to focus students' attention on what they belic\ed 
was helpful or was in need of change m regards to specific interview components (c.g.. note's taken 
during quiet lime, icflee live responses pio\ided by intcrv iewer. concept map), and asked them to 
provide reasons foi their answeis. Students' responses here weie pooled wiih responses t o the first 
question to determine the prevalence with which all POSTIMCl students perceived specific interview 
components as helpful. This pooled data revealed that more students (760 t reported concept mapping 
to he helpful than any other interview component. Reasons provided hv the students suggested furl her 
the utility of the concept map in fostering mclaeognilion and triggering recall ol what they had learned 
about CT(V For example. Roberta speaks to the metaeugnilive nature of the concept map as site 
idem ifies it as an image she used to help her monitor and check on the completeness of her recall of 
knowledge: 



Rohcita: Well then. I mean when you look at the map and then you can always find something 

you missed. If you go over it a second lime, then von are usually able to think 
through vour memoiv and make sure you did not miss anything. 

Knl also talked about the map as an image to laei lit ato recall: 

Lari: |.I|itst looking at it and seeing all the words helped me to think of the other terms. 

A post hoc inspection bx the interviewer ol the students’ notes, jotted down by the student during 
the "quiet time." repealed that four (2dH ) students chose to sketch a concept map (as opposed to just 
witting notes) as a vehicle to lecord some of their thoughts during the quiet time. The interviewer did 
not cue anv students about using a concept map as one option for recording their thoughts during the 
quiet lime. He simpK stated that the notes might be words, phrases, a diagram-- most important was to 
make the notes "useful to von." However, perhaps the interviewer's mention that students would be 
toiisliuelmg a comcpi map dm mg the iniciview stimulated this action. Alternatively, perhaps thinking 
about Cl Vs brought to mind images ol concept maps, or subsets ot concept relationships explicated 
t herein, that had comprised a portion of the instruct ion on (TVs. Regardless, this choice speaks to the 
value that these students placed on the concept map as a means to help them think and remember about 
('ICs Interesting!} a post hoe inspection of the POST ICC I students' "quiet time” notes also revealed 
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dial three ( I 8*4 ) students had chosen to “concept map” theii miles. Figure v ) lilustiaies the cuneepi map 
notes' prepared by one of these students. In the investigator’s opinion, it is unlikels that an eighth 
grade student could have eonstmcled " Iroivi scratch*' this entire concept map during the brief oh minutes, 
quiet lime period, without utilizing mental images of some of the node-link-node networks explicated 
therein. Accordingly, the investigator believes that components of this student s map had been j maced 
by the student Irom science instruction on Ci : ( This may have been the case will; the other two 
POSTK'CI students who drew concept maps during the quiet note-taking time. One nl these other 
students. Ariel, had this to say about the utility ol concept maps; 



Interviewer Win there anything that comes to mind that helped you remember, think, or i a I k 
about (T‘(\ (in this interview]? Anything that you did or I did? 

Ariel The concept maps that Mrs. [my teacher] gave us. lake and -die kept telling us m 

go over them and do them and do all these for homework. Si 1 ), that kind ol helped 
me. . . So. that’s why like I did t hat [drew the concept map toi no n< *tes| . cause ! 
remember. And that's what I did my test on. 

Ariel s comment suggests she may have imaged some concept maps fiom her science class 



Insert Figure 9 here 

The remainder of the inleivicvv questions for the POSTIMOI students hnned-in-on the unwept 
map. in order to elicit lurther comments on this interview component and to ascertain if and how ihe 
concept map might have allcctcdthe students answers and thinking to the interview questions. Si\i\- 
li\e {(oi percent of the students said that they believed their answers to the interview questions would 
have been different than they gave today had the concept mapping not been a component ol the 
interview. Conversely. 2 ( Pr stated theii answeis would have been the same Participants geueiallv uc:c 
not emphatic in responding one way oi the other. One student. Samuel, who said his answers would 
have been soil of ” the same, added this; 



Samuel; Well. 1 still. I knov\ in my head but I think that (concept map] helps me see it and Im.v 
they relate to each other. 

l ike Samuel, some o| the students who believed their answers were diffeicnt due to the concept 
mapping interred that the visual nature of the map was an asset to them while thinking about oi 
responding to the inleivicvv questions. Mollie put it this way: 
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Muliie: [Ij could look at what 1 had said and so 1 could t. -member vvluit I didn't say and . . . so 

I wouldn't leave anything rut. 

Julia provided evidence of her melaeognitive knowledge in referring to the visual nature of' the map as a 
helper in explaining her answers: 

Julia: It’s easier for me to look at something and explain it than too just think off the top o! 

my head. 

Other of these students gave ev idenee that the visual nature of the map triggered in them ti.e spreading 
activation process Curley serves as an example: 

Carlev : Cm. I d look at one piece of paper— it'd make me think of something else. 

The students who believed their answers differed due to the concept map often attributed this difference 
to the fact that the map nr mapping process refreshed their memory or simply helped them to remember. 
Here, some students referred to the value of the concept connections explicated in the map and that this 
feature of the concept map was a valuable addition to the notes they had taken during quiet lime. 

I aieille's response serves to illustrate this: 



Lucille: W ell, whenever we made the concept map. it kind of let me think about what, what I 

could like connect things from. And that kind of helped me remember what to 
answer the questions— if I had forgotten some things. 



Most ol the students w ho believed that the concept mapping component of the interview did not 
change their answers to the interview questions stated that the map had no influence on their thinking 
These individuals saw the mapping as just restating what they had verbalized, oi that the notes they had 
taken dm mg quiet time had the same influence as did the concept mapping on theii answers. 

One student suggested that the interviewer could consider giving the students a word list to use 
m constructing the map. Another studc.it wanted to have a fill-tiMhe-blank concept map instead of 
building one fiom seiateh. This student suggested the following as a iccommendcd modification to the 
concept mapping procedmcs employed in the interviews: 

Javk: Well. I think like you should have find out what wo are doing in science and have 

like maybe a review thing to help you remember everything. Lake make a review 
concept map wheic you fill in the "blanks, instead of just making up your own from 
scratch. 
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Jack's suggestion paralleled some of instnielionul activities he engaged in to build eoinpeicnee in 
concept mapping and in studying material from the STS-GAC' curriculum, as revealed by the interviews 
w ith his teacher: 



Interviewer: 


Now. if you would please tell me, for the concept mapping . . how sou developed 
their lynur students'! competence in it. 


Teacher. 


And then I started giving them— at the end of the [science ]unit--I would give them 
a concept map with boxes with connectors in place, some terms in place, and then 
a list of terms ai the bottom that they needed to pul in the appropriate box to show 
that they understood the connection. . . .Then we graduated from that to some 
with just the boxes filled in and they did the connectors. And then ultimatek to 
just, "Here is a list of terms that you should be able to connect." 


Interviewer: 


Now , in the GAC' unit, in general to this point, could you just qualitatively 
desc ribe the extent of your use of concept mapping. For example, students 
view ing concept maps, students making concept maps. 


feacher: 


1 took a number ol the ones that were in the teaching materials and whited out 
boxes and alter we had gone over a lesson, gave it to them and said. "This is a 
summary of the lesson and you go back and fill in the boxes". . . So. 1 felt for the 
kids that had diflieulty taking notes, as many eighth graders do, that that was there 
back up. That that became their security blanket. 



I he S I S -( i At. curriculum and guide on concept mapping that the investigate provided to the teacher 
did not suggest this particular application of the concept map tool, however, the investigator has 
observed other curricula that employ concept maps in this way. The excerpt that follows from the 
interview with Carles suggests that students did hone-in-on these maps in studying about CFCs and their 
role in GAC. 



Inicrvicwci : 


Was tlieie anything he real frank svith me. was there anything that was helpful? 
That is. helped you remember, think and talk about CFC’s? 


Carle} : 


f he concept map helps. 


Inters iew er: 


Okay noss. foi what reason was that helpful to you? 


Carles': 


1 *m. like at night lime sshen we’re studying this. I’d go home and I'd study it. And 
it svas a lot easier than to have to read you knosv. the chapters in the hooks and 
si lift . 



C ai ley s comment suggests further that she \ allied the concept map as a study aid. It is likely this \ aluc 
also was held by various other of her classmates. Evidence for this comes from comments made In her 
■eacher during, the teacher inters lews that followed the completion of instruction: 



Teachei : 



Ami todav. we skilled a new chapter on simple machines, anti that was the first 
thine they said Cause I said, “Gel out paper we are going to take some notes." 
They said: "We going to do a concept map?" 1 Laughter] And I said, ' Not right 
now, we are just going to go with some straight notes. " 

Interviewer: Do you sense that the majority of them enjoy the concept mapping or not really? 

Or they seem to ah. you said they ask you 

Teacher: Yeah, they seem to enjoy it. They seem to understand it . . . 

The investigator also engaged in conversations with the teacher after data analysis, in order to 
verily further his contention that the students valued concept mapping. Here, the teacher revealed that 
over one-half of the students w ho participated in the study went on to develop on then* own ( i.e., 
unsolicited by the teacher) concept maps for subsequent units of study in science class. Further, the 
teacher reported that often these students chose to use such maps as opposed materials developed by the 
text publisher to stiu.lv from for science unit examinations. 

Conclusions and Recommendations 

lo the investigators’ knowledge, no othei studies have employed concept mapping as an 
interview tool in the manner reported above. Therefore, the conclusions that follow are tentative in 
nature, and warrant further research In this study, an interview that embedded a concept mapping 
process (compared to an interview that excluded this process) did not effect statistically significant (p < 
.05 ) changes in the oxlcrnali/ation of students' conceptual understandings. Considering the potential 
impact (Mi cognition of the concept map tool, the investigators' were surprised that the concept mapping 
process failed to enhance the cxiernaii/ation of students' conceptual understandings. The concept map 
piovides a graphic visual of structural knowledge that, in theory, extends working memory (Johnson <kr 
Thomas. 1992: Jotiassen et ah. 1095). As such, it seems logical that it would trigger spread-of-ucti vation 
and enhance recall ( Ausubel et al.. I97X) Fuithermore. the act of concept mapping should force critical 
thinking (Novak <k Cumin. I9S--I), and aecoidingly. enhance the externali/.ation of interrelationships 
amongst cone *pts. The concept map as evolving product and process in the interview setting also 
■should enhance metacogmtion (Flavell. 19X5: Novak. 1990b: Wandetsee. 1990). 

The investigators conclude that the tollow mg contributed to the I Hiding that the concept 
mapping component of the post-instruction interviews had no additive effect on students’ recall and 
explication : ia) The quiet time interview component, offered to students prior to the posing of the 
interview questions, facilitated a stale of “mental readiness” and likely enhanced the externali/.ation of 
ill students posi-insirtMional understandings: ib) The interview topic- -CO CA and their role in GAC-- 
w in ioo | mined a locus, and as such, did not stimulate or allow for much elaboration; (e) I extensive 
exposure to concept mapping during STS GAG instruction resulted in students learning vvcll- 
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anchoring in semantic memory (Johnson & 1 liomas. 1992)--nutterial on (..'PC's and t Me i r role in CJAC. 
and portions of coneept maps from instruction may have been imaged (West el ul.. 1991 ) 

Another factor that might explain the.se non-significant findings deals with the le\els of cognitive 
outcomes associated with the subject matter. Perhaps the interview questions used to elicit conceptual 
understandings did not engage sufficiently thinking at the application through evaluation levels, and as 
such, concept mapping offered no advantage over knowledge that was available due to rote learning 
(Novak. 1990b). Schmid and Telaro ( 1990) found that the instructional strategy of concept manning 
significantly enhanced performance only on application and analysis level questions. 

Other reasons that may have accounted for these findings of non-significance were that (a) 
concept mapping was still a lelat i vely new strategy lor these eighth grade students, and (b) such students 
are used to rote learning (Novak. 1990b). Novak contends "Junior high school students have become 
adapted to primarily lote-mode learning and it is not easy to move them to meaningful learning 
strategies" ' p . 41 1 . It is possible that the findings from the interviews may have been different if 
students would have been involved in constructing their own concept maps over a longer instiuctional 
period. Finally, given their exposure during instruction to several concept maps of the “instructor- 
prepared’' variety, it is possible that some students had come to view concept maps on GAC as being 
somewhat static-more as finite entities than as a tool for elaborating their knowledge. Schmid and 
lelaro ( 1990) suggest that the use ol teacher-prepared maps w ill result in "the same kind of rote 
memorization already so common in the schools" (p. 84). However, the meta-analysis of concept 
mapping studies by Horton et al. 1 1993) do not support this contention- -they concluded that student 
prepared concept maps were not superior to instructor prepared maps in effecting student learning gains. 

Despite the fact that TYPF1TV was not a predictor of any of the students' measures of conceptual 
ur.dei standings the interview component perceived as helpful by the most POSTIMCI students was 
concept mapping. The reasons students provided as to why the concept mapping was helpful indicated 
that they used the map metaeognitively and that ii triggered the spread-of-activation process. 
Additionally, the majouiv ot POSTIMCI students believed that the coneept mapping component ot the 
interview resulted in their answers to the interview questions being different— and inferred the mapping 
had a positive impact -lrom what they would have been had concept mapping been omitted from the 
interview Post hoe analyses, where gender and ability level were regressed on categorical predictors 
coded to reflect the presence or absence of these perceptions (concept map helped or resulted in answ ers 
being different), revealed that these perceptions cut across ability level and gender (Rye. 1995). 

The investigators recommend that future research should examine the effectiveness of the 
Pt JSTIMCI protocol w ith students w ho are no! exposed to any concept maps of the “instructor- 
prepared'' variety, and peihaps not even assigned any concept mapping activities, during instruction on 
the topic tor which understandings aic being examined. This condition would necessitate the 
development and verification of student competence in concept mapping p rior to such instruction. Ideal 
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conditions in such a stud) might also include the billowing: iu) student^ would have considerable 
prc\ ious experience in developing coiicepi maps). < b ) concept mapping competence \\ miki be verified 
through a task developed and assessed by the investigator, (c) the interview topic would have greater 
breadth than that in this study, (d) the mteiview questions would engage more so the application and 
analysis levels of cognition, and (ei “quid time" would be examined as an independent variable, i.e.. a 
4-group design would be employed to allow for the examination of tiny separate impacts of "type of 
interview" (i.e. with or without a concept mapping process) and "quiet time" on the extcrnali/.ation of 
students' conceptual understandings. In regards to the exclusion of explicit illustrations or assignments 
that involve the concept map tool, once students are competent in concept mapping, the findings of tins 
stud\ suggest that students will employ concept mapping on their own. Therefore, the absence during 
instruction of such visuals or assignments is no guarantee that instruction will have excluded exposure to 
concept mapping. 

The measure of similarity between student and teacher Pathfinder Netv\orks--PFNSl— was a 
worthwhile addition to the two-parameter regression model for predicting the measure of accordance 
(ACCORD). PFNSI had predictive validity (Goldsmith et ah. 1991) for performance in the interview on 
the measure of ACCORD, and proved to be a reliable confirmatory measure (Jonassen. 1993) of the 
degree to which students held an ideal post-instructional understanding. The predictive validity of 
PFNSI for ACCORD is considered quiet high, given that only X concepts formed the basis of the 
concept relatediicss ratings that were transformed into Pathfinder Networks, and ultimately to the 
measure of PFNSI (Goldsmith et al.. 1991 ). The investigators infer that the knowledge engineering 
process of constructing an expert concept map vv uh the teacher facilitated the selection ot concepts tor 
Palhfindei analysis that, in the form of concept (elatedness ratings, were associated with substantial 
predictive power. 

More research is needed that incorporates the concept map and Pathfindci Network tools, 
kleallv. such investigations would include a pre- to post-instruction comparison and assessments against 
expert leferent structures. This would make a contribution to the research base that shows students' 
know ledge structures become more like their instructors and/or other experts as they develop expertise 
iGuldsiniih A Davenport. 1990). Complementary to this would he research that attempts to replicate or 
expand the findings of this study, where an expert concept map is utili/ed to identify domain-specific 
concepts for concept relatedness ratings, and these ratings are transformed into Pathfinder Nelw ork 
know ledge measures ic.g.. PFNSI ) and regressed on performance measures to determine predictive 
validitv Further, such research could build upon that reported by Acton el al. ( 1994) by employing 
more than one expert referent for determining PFNSI. and examining whether or not referents other than 
the instructor resulted m gtealer predictive validitv for PFNSI. 




•> i 

tj i 



References 



Acton. \\ .. Johnson, P.. (Sr Goldsmith. I . ( 1994). Structural knowledge asscssitic nt: Coinparison of 
referent structures. Journal of Educa ti onal Psychology . 86, 303-311. 

Auld. G 1 1990). Adult gender differences in epistemology and knowledge about fat and cholesterol. 
Disser tati on A bstracts Int erna tio nal . 5_U 2824. 

Au.subel, D.. Novak. J., A: Hanesian, H. (1978). Educationa l psycholo gy. New York: Holt. Rinehart 
ami Winston. Inc. ' 

Brody. M. 1 1991 ). Understanding of pollution among 4th. 8th. and 1 Ith grade students. J ourna l > a' 
Environmen tal Educa tion . 22. 24-33. 

Cohen, \1. ( 1 99 1 ). Cognitive factors and their role in the enhancement of the knowledge elicitation 
process (Expert s\ stems. Mvers Briggs type indicator). Disserta tion Abstracts International. 52. 
1695. 

Cordinglex. E. ( 1989). Knowledge elicitation techniques for knowledge-based systems. In Diaper, D. 
(Ed.), Knowledge elicitatio n (pp. 89-176). Chichester. England: Ellis Horwood Limited. 

CTB McGraw-Hill. (1986;. CAT. California Achievement Tests Forms h A: f. Nouns B(>ok Dec - 
Feb. Monterey. CA: CTB McGraxv Hill. 

Dana. N , Dana. T.. Kelsay. K.. Thomas. 1) . & Tippinx, D. ( 1992, January). Qualitative interviewing 
and the art of questioning: promises, possibilities, problems, and pitfalls. Paper presented at the 
1992 Qualitative Research in Education Conference. Athens. GA. 

Driven R. ( 1989). Students’ conceptions and the learning of science. Internati onal J ourn al of Science 
Edu cation. 1 1 . 481-490. 

Ennis. C\. A: Marcus N. (1994). Bi olog ical consequences of global climate change. Boulder. CO: 
University Corpontlion for Atmospheiic Research. 

l.rlandson. D.. Harris. F... Skipper. B.. A: Allen. S. i 1993' Doi ng na turalistic inquiry. London: Sage 
Publications. 

Ex Ion. B., A: Linn. M.C. (1988). Learning and instruction: An examination of lour research 
perspectives in science education. Revievv of Educational Resea rch. 58. 251-301. 

Lirlej. M.. A: Hellens. D ( 1991 ). Knowledge elicitatio n. New York: Prentice Hall. 

Flavell. J.H. (1985). Cogn itixc de velo pment.. E.nglexvood Cliffs. NJ: Prentice -Hall. Inc. 

Gagne. F.. ( 1985). T!ie eo gnilixe psychology of scho ol learning . Boston: Elide. Biown. and Company 

Goldsmith T. A: Davenport, D. (1990). Assessing structural similarity of graphs In Sehvaneveldt. R. 
(F.d.i. Path fin der asso ciati v e networks (pp. 31-52). Norwood. NJ: ABEEX Publishing 
Corporation. 

Goldsmith. T.. Johnson. P.. A: Acton. \\ ( 1991 ). Assessing structural knowledge, lournal of 

LUuc at i qnai j 3 syc l/iol ogy, 83. 88-96. 




30 






Good, R.. No\ iik . J. ' ’andcisee. J. *Lds. ) i 1990). Special issue: Pei's pec U\ es on concept mapping. 
Journ al of Res can, n in.Scjejiye Tcychijig^ J7 ( 10) 

Gordon. S. & Ciill. R i 1989). Question probes: A structured method fur eliciting declarative 
knowledge. A1 Applica tions in Natural R esource Mana gement. 3. 1 3-20. 

Gordon. S.. Sehmierer. K.. <Y Ciill, R. ( 1993). Conceptual graph. analysis: Knowledge acquisition for 
instinct tonal system design. Human Factors, 35. 459-481. 

Hcin/e-Fi\. J. A. ( 1987). Evaluation of concept mapping as a tool for meaningful Ldncalion of college 
biology students. Unpublished doctoral dissertation, Cornell University. NY 

Hem/c-I r\ J. Novak. J. ( 1990), Concept mapping brings long-term movement tovvanl meaningful 
learning. Science F.dueation. 7.4.461-472, 

Horton. P. B.. MeConney, A., Gallo. YE. Woods, A.. Senn, G., & Ilamelin, D. (1993). An investigation 
of the effectiveness of concept mapping as an instructional tool. Scie nce Edu cat ion. 77 . 95-1 1 l. 

Houghton. J. 1'.. Cailendar. B. A.. A: Varney. S. K. tEds.) t 1992). Clim at e change 1 992. The 

supple mentary repo rt to the 1PCC scientific a ssessment , (pp. 1-22). New York: Press Syndicate of 
the University of Cambridge. 

Johnson, S. & Thomas. R. ( 1992). Technology education and the cognitive revolution. The 
Tech nology Teac her. 5 1 . 7-12. 

Jonassen. D. { 1996). Comp uters in the classr oom : Mindtools for critical thinking. Emilevvood Cliffs, 
NJ: Prentice-Hall. 

Jonassen. D. 1 1993). Changes in knowledge structures from budding semantic net versus production 
rule representations i»f subject content. Journal o " C omp uter -Bas ed Instruction. 20, 99-106. 

Jonassen. D.. Bcissner. K &Yacci. Nl. ( 1993). Struct ural know ledge. Hillsdale. NJ: Lawrence 
Hilbaum Associates. Publishers. 

Judd. ( . M., & McClellan. G. 11 1 1989i. D ata an al ysis : A models a pp roach. Florida: fiarcourt Brace 
Jen anovich. 

1 a! ranee, M. ( 1992). Questioning knowledge acquisition. In T. 1 aucr. I:. . Peacock. K A. ( uacssei 
( Hds. ), Questions and tn.lqriijatjiin wstuiis tpp. i I 28). Hillsdale. NJ: Lawrence Frlhaum 
Associates, Inc. 

1 .omask, M.. Baron. J.. tY Grieg. J ( 1993. August). Assessing conceptual understanding m science 
through the list' of two- and three* dimensional concept maps. Paper presented at the Third 
International Seminar on Misconception^ and Educational Strategics in Science and Mathematics. 
Cornell University, Ithaca. NY. 

Markham. K.. Mint/cs. J.. and Jones. M. Ci. ( 1994). 4 he concept map .is a reseat ch and evaluation loop 
Further ev ulencc of v alidity. Journal of Re search in.ScuMKc Teaching, 3J . 9 1 101 . 

Marshall. C.. A Rossnian G <1988). Designing q ua litative res earc h. London: Sage Publications. 

Meuium. S. < 1988c (Vise study research in education: A qu alitative approach. San Francisco. CA. 
Jnssev-Bass Inc. 



Nhkhleh. M. t I WO). A study o! students’ thought processes arid understanding of acid/base concept 
during the performance of instrument based titrations. Dissertation Abstracts Inte rnatio nal. 52. 431 

Novak. J. ( 1992). A view on the current status of Ausubel's assimilation theory of learning. A paper 
presented at the meetings of the American Educational Research Association. San Francisco, C A. 

Novak, J. ( 1990a). Concept mapping: A useful tool for science education. J ournal of Resear ch in 
Science Teaching . 27 . 937-950. 

Novak, J. ( 1990b). Concept maps and vee diagrams: Two metucognilivc tools to facilitate meaningful 
learning. In struc tional S cien ce, JJ), 29-52. 

Novak. J. { 1985). Metalearning and metaknowledge strategies to help students learn how to learn in 
L.H. West & A. E. Pines (Eds.). Cognitive structur e and co nceptual cha nge (pp. 101 -I 15). 
Orlando. FL: Academic Press. Inc. 

Novak. J. <fc Musonda. D. ( 1991 ). A twelve-year longitudinal study of science concept learning. 
American E du cational. Research Jojurnal, 28. 1 17-153. 

Novak. J. Govvin. D. t 1984). Learn ing how to learn New York: Cambridge University Press. 

Okehukola. P. A { 1992 ». Attitude of teachers towards concept mapping and Vce diagramming as meta- 
learning tools in science and mathematics. Educational R esearch , 34, 201-213. 

Osborne. R. & Freybcrg. P. (F:ds.) 1 1985). Learning in science . Auckland. New Zealand: Heinemann 
Education. 

Pattern. M. (1990). Qualitative evaluation and r cse aic h methods, l.otulon: Sage Publications. 

Pidgeon. N.. Turner. FL. & Bleckley. D. ( 1991). The use of Grounded Theory for conceptual analysis in 
knowledge elicitation. International Journal of Man -Ma chin e Studies, 35. 151-173. 

Posner. G.. <k Gert/og. W ( 1982). The clinical interview and the measurement ol conceptual change. 
S cien ce Ed ucati on, 06, 195-209. 

Rubba. P.. <N Wicsonma\cr. R. \ 19S5). A goal structure for precollege STS education: A proposal 
based upon recent literature in environmental education. Bulletin o f S cien ce, Tec hn ology and 
Society, 5. 573-580. 

Rubba, P., Wiescmnaycr. R.. Ditty, T. <V Rye. J. 1 1996). The Leadership Institute in STS Education: A 
collaborative teacher enhancement, curriculum development and research project of Penn State 
University and West Virginia University with rural middle/junior high school science teachers, 
journa l o f Science Tea che r E ducat ion. 7. 23-40 

Rubba. P.. Weisenmaver. R.. & Rye. J. el al . (Hds) ( 1995). Global a tm ospheric change: The 

enhanced greenhouse effect., ozone layer depiction an d o/.one p ollution. I T ni \ ersity Park. PA: lhe 
Pennsylvania State University- 

Rye. J.. Rubba. P.. <V Wiesenmayer. R. (In press). An investigation of middle school students' 

alternative conceptions of global warming as formative evaluation of teacher-developed STS units. 
Inte rnation al Journal o_f Science HducaUun. 

Rve. J. i 1995) AiLl.ipesUgat.ion of. J be. On.ieept.Mup as an Inte rview Tool t o Fac ilitate Ex lcrnali/at ion 
oKjlheepii':*.! [ Viders landings associated with Global atmospheric Change by Eighth Grade 




32 



Physical Sc ience St udents . (.‘npuhlishcd doctoral dissertation. The Pennsylvania State University, 
University Park. 

Rye, J.. Rubba. P.. & Wiesenniayer. R. (1994, March). Middle school students’ conceptions of global 
warming following STS instruction. Paper presented at the 1994 Annual Meeting of the National 
Association for Research in Science Teaching, Anaheim, CA. 

Schmid, R., & Telaro. G. (1990). Concept mapping as an instructional strategy for high school 
Biology. Journal of Educat iona l Res earc h. 84. 78-85. 

Sehvancveldt. R. (Ed.) (1990). Pathiinde r associat ive net works . Norwood. NJ: ABi-HX Publishing 
Corporation. 

Shuell, T. ( 1985). Knowledge representation, cognitive structure, and school learning: A historical 
perspective. In L. H. West & A. L. Pines (Kds.). Co gnitive str uc ture and conceptual change (pp. 
101-115). Orlando. FL: Academic Press, Inc. 

Smith. K. ( 1987) Educational engineering: Heuristics for improving learning effectiveness and 
efficiency. Engineering Education, 77 . 274-279. 

Spradely. J. (1979). The ethnographic intervie w. Orlando. FL: Holt. Rinehart and Winston. Inc. 

Stensvold. M. & Wilson, J. (1990). The interaction of verbal ability with concept mapping in learning 
from a chemistry laboratory activity. Science Education . 74 . 473-480 

Swaffield, G. ( 1990). Development of a structured method for knowledge elicitation. Dissertation 
Abstracts International, 51 . 3104. 

Wallace. J. & Mint/.es, J. (1990). The concept map as a research tool: exploring conceptual change in 
biology. Journal of Research in Science Teaching. 27 . 1033-1052. 

Wandersee. J. (1990). Concept mapping and the cartography of cognition. Journal of Research in 
Science Teaching . 27 . 923-936. 

Wandersee, J.. Mint/.es, J., & Novak. J. ( 1994). Research on alternative conceptions in science. In D. 
Gabel (Ed.). Handbook of researc h on science teaching an d learning (pp 177-210). New York: 
MacMillan Publishing Company. 

West. C.. Farmer. J., & Wolff. P. (1991 ). Instruction de s ign: Implications from cognitive science. 
Englewood Cliffs. NJ: Prentice Hall. 

White, R.T. (1985). Interview protocols and dimensions of cognitive structure. In L.H. W'est & A L. 
Pines (Eds.). Cognitive str uctu re an d conceptual change (pp. 51-58). Orlando, FL: Academic 
Press. Inc. 

White, R.. & Gunstone, R. (1992). Pr obing understanding. New York: The Fulmer Press. 

Wilson. J. ( 1994). Network representations of knowledge about chemical equilibrium; Variations with 
achievement. Journal of R ese arch in Science T eaching . 31 . 1133-1147. 





33 



GA 



(b O4 



XfaSrOi Xi R a 



GB O3 O4 



K ey to Symbols 

X fa = Instruction of students from foundation/awareness lessons. 

S r = Student recruitment for study. 

O] - Concept interviews with students prior to investigation lessons. 

Xj = Instruction of students from STS-CiAC investigation lessons. 

Ra = Random assignment of students completing Oj and Xj to GA or GB. 

GA - Group of students to undergo, after investigation lessons, a concept interview 
that would not embed a concept mapping process. 

GB =5 Group of students to undergo, after investigation lessons, a concept interview 
that would embed a concept mapping process. 

02 = Concept interviews with students in GA after investigation instruction. 

03 = Concept interviews with students in GB after investigation instruction. 

04 = Relatedness ratings of GAC concept pairs. 

Fi gure 1 . Research design 
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Figure 3 Teacher-expert Pathfinder Network 
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Figure 4. Pathfinder Network of the student with the highest Pf-'NS] 




Figure 5. Pathfinder Network of the student with the lowest PI NSI 
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Fi gure 6 . One of the least complex concept maps emergent from POSTIMCI 
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Figu re 7. One of the most complex concept maps emergent Imm POSTIMCI 
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Figure 8 , Hybrid-type concept map emergent from POSTIMCI 




F igure n. (.'(incept map "notes" drawn during quiet time by POS’I ICO student 
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Table 1 



Questions to Elicit Students’ Conceptual Understandings about CFCs in All Interviews 



1 In your thinking, what are CFCs? 

2 Where do CFCs come from? 

3 What, if any, problem is caused by CFCs? 

3a How do CFC’s cause (problem cited by student) ? 

3b I believe ( problem cited bv student ) is a problem because. 

4 Now, is there anything else you remember or would like to tell me about CFCs? 

3 What else, if anything, would you like to tell me about any problems caused by CFCs? 





Table 2 



Questions in Post Instruction Concept Interviews to Elicit Students' Views on the Interview as a Means to 
"Tell What You Know" 

Questions Posed to All Students 

1 What are your views and opinions about this interview as a way to help YOU think and talk about 
C'F C s? 

2 I am interested in things you found helpful in the interview as well as things you believe should be 
changed or added to make the interview more helpful. 

(a) What, if anything, do you believe was helpful? 

(b) For what reason or reasons was ( j helpful? 

(e) What, if anything, do you believe needs to be changed or added .’ 

(d) For what reason or reasons should I change or add ( _ i? 

Additional Questions Posed to Students Undemoine PQSTIMCI 

3 W hat, if any, (other) comments do you have about the concept mapping that you did as a part of this 
interview? 

4 Imagine that you went through this same interview , but without doing the concept mapping'. 

(a) Do you believe > our answers to the questions about CFCs would have been the same as you gave 
me today, or would your answers have been different? 

( b) Please tell me your reasons for saving this ? 

(c) It student responds different to (a), ask: In what wavs would your answers have been different it 
we had not done the concept mapping? 

b hi what ways, if at all. did you make use ol the concept map during the interview .’ 

6 How, if at all, did the concept map influence your thinking to the interview questions about CF-Cs? 

7 In your opinion, was drawing anil having the map helpful or not helpful to you in tellinc me what vou 
know about CF Cs? 

f a) If students says "helpful ask: For what reason or reasons was having the map helpful? 

(b) If student says "not helpful” ask: What, if any. ideas do you have that WOUL.D make 
dru wing/having the map helpful? 
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Table 3 



Descriptive Statistics for Result 
Conceptual Understanding. 


sof POSTICC1 and POSTIMC 


I on Dope 


ndent Variable 


Measures of 






POSTICCI 

01=17) 






POSTIMC1 
_in 5.121 




Dependent 












Variable 
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! 

i 

| 

\7? 

rl 


Mean 


Sid Dev 


Range 


Mean 


Std De\ 


Accordance 














ACCOR DC 


6- 2b 


1.X.K9 


6.07 


4-2X 


16.71 


6.22 


ACCORDR 


1-24 


14.X4 


6.24 


2-24 


14.00 


6.04 


ACCORD 


7 44 


33.77 


12.07 


6-44 


30.7 i 


12.14 


l 7 sternal 














Related ness 














HX1 HR. NT' 


117 


4.06 


3.6X 


0-13 


3.35 


3.00 


HXTHRNR 


1-12 


3 65 


2 74 


0-12 


TOO 


3.15 


HXTF-.RN 


2-24 


7.71 


6.22 


0-25 


6.41 


6.02 


Interrelatedness 














1NTHR 


0.24-1.04 


0.74 


0.17 


0.5-1 


0.X2 


0.16 



Xnte. A(. CORD and HXTfiRN represent the sum of the concepts (ACCORDC or KXThRNO and 
relationships (ACOORDR or BXTFRNR ! explicated by each student. 




